Compression techniques such as pruning and quantization offer a solution for more efficient deployment of language models (LMs), albeit with small performance drops in benchmark performance. However, general-purpose LM compression methods can negatively affect performance in specialized domains (e.g. biomedical or legal). Recent work has sought to address this, yet requires computationally expensive full-parameter fine-tuning. To this end, we propose cross-calibration, a novel training-free approach for improving the domain performance of compressed LMs. Our approach effectively leverages Hessian-based sensitivity to identify weights that are influential for both in-domain and general performance. Through extensive experimentation, we demonstrate that cross-calibration substantially outperforms existing approaches on domain-specific tasks, without compromising general performance. Notably, these gains come without additional computational overhead, displaying remarkable potential towards extracting domain-specialized compressed models from general-purpose LMs.

By submitting this content request, I have legitimate interest in the content and agree that Technology Innovation Institute, their partners, and the creators of any other content I have selected may contact me regarding news, products, and services that may be of interest to me. By submitting this content request, I have legitimate interest in the content and agree that Technology Innovation Institute, their partners, and the creators of any other content I have selected may contact me regarding news, products, and services that may be of interest to me. I agree to the IEEE Privacy Policy Are you an IEEE member?

A machine learning program has been used to distinguish between cancerous and healthy cells by identifying patterns in characteristic combinations of genes.

By registering, you agree to the AWS Event Terms and Conditions and the AWS Community Codes of Conduct. By completing this form, I agree that I'd like to receive information from Amazon Web Services, Inc. and its affiliates related to AWS services, events and special offers, and my AWS needs by email and post. You may unsubscribe at any time by following the instructions in the communications received. By completing this form, I agree that I'd like to receive information from Amazon Web Services, Inc. and its affiliates related to AWS services, events and special offers, and my AWS needs by email and post. You may unsubscribe at any time by following the instructions in the communications received.